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Overview 



Performance-based scholarships are a new and innovative type of financial aid for postsecondary 
education. The scholarships are performance-based in that the payments are contingent upon meeting 
academic benchmarks after students enroll in the scholarship program, and do not place weight on 
past grades. This design is intended to give students an incentive to change their behavior with respect 
to time management and academics, with the goal of improving their educational outcomes. It also 
serves students who may not have had much past academic success, and gives them a clean slate. 
Researchers, institutions of higher learning, and policy experts have a growing interest in performance- 
based scholarships as a tool to promote academic success, particularly for low-income populations. 

Through the national Performance-Based Scholarship (PBS) Demonstration, MDRC is testing 
variations of this intervention to build evidence on its potential to help low-income students. Ohio is 
one of six states in the demonstration, and is unique in that its program had existed independent of 
MDRC’s research. Ohio’s history of cultivating political will to finance a performance-based 
scholarship with public funds makes this variation especially policy-relevant. The program was 
offered to low-income parents at three community colleges. This population may struggle with many 
competing priorities, such as child care or work, in addition to college. 

The PBS Demonstration uses random assignment, which is the gold standard in program evaluation 
and the research methodology preferred by the Office of Management and Budget for demonstrating 
program effectiveness. MDRC assigned about 2,300 students to either a control group that received 
the colleges’ standard financial aid package, or to a program group that received the same aid in 
addition to becoming eligible for receipt of the scholarship. The program lasted for one academic year 
and offered students awards up to $1,800 per academic year for earning a “C” or better in 12 or more 
credits per term, or up to $900 for meeting that benchmark in 6 to 11 credits per term. Students were 
paid at the end of each term, contingent on meeting the performance benchmark. 

This report covers the full program year for the first cohort of the study, which comprises students 
who enrolled in the study in the fall 2008 term. The study sample for the fall 2008 cohort includes 
about 1,300 students, or roughly 60 percent of the total sample. Early analyses show that: 

• The program increased the number of credits attempted. Program group students at- 
tempted more credits on average than the control group students in the second program term. 

• The program increased full-time enrollment. Program group students were 13 percent 
more likely to enroll full time than control group students in the second program term. 

• The program increased the number of credits earned. Program group students earned 
more credits in both the first and second program terms, and earned an average of two full 
credits more than control group students over the program year. 

• The program reduced educational debt. Loans made up a smaller proportion of total fi- 
nancial aid for program group students than for control group students. On average, program 
group students were awarded $237 fewer loan dollars. 

• The program has not had any impact on persistence to date. The control group students 
registered in the second program term at a high rate that was difficult to improve upon. 
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Preface 



Can supplemental financial aid that is tied to academic performance help students complete their 
college studies? Although federal and state financial aid can help with college expenses, students still 
often have unmet need, particularly if they are from the poorest families or are independent of their 
parents 8 or are parents themselves. Such students often get jobs to supplement their income, but too 
many hours on the job can hurt their academic performance and they may drop out of school. Loans 
can also help, but many low-income students are reluctant to take on debt 8 especially if they have 
doubts about their ability to earn a degree. 

As part of its multisite Opening Doors demonstration, which sought to confront some of these 
problems head-on, MDRC evaluated an innovative strategy: granting performance-based scholar- 
ships. The idea is to increase financial support for students and create an incentive to complete their 
courses, which in turn should accelerate then progress toward attaining a degree. Promising results 
for full-time registration, persistence, and credit accumulation from the Opening Doors program in 
Louisiana spurred interest among other states in offering similar scholarships. Recognizing, however, 
that Louisiana is just one study and that the results may not be easily generalized to other populations, 
MDRC launched the Performance-Based Scholarship Demonstration in 2008 with anchor support 
from the Bill & Melinda Gates Foundation. The demonstration is testing variations of a PBS program 
in Ohio, California, Arizona, New Mexico, Florida, and New York, evaluating whether such scholar- 
ships increase academic achievement. 

Students at three community colleges in Ohio were assigned to a control group that received the 
collegesOstandard financial aid package, or to a program group that received the same aid and were 
eligible to receive the scholarship. The program offered awards up to $1,800 per academic year for 
earning a ft 6 or better in 12 or more credits per term, or up to $900 for meeting that benchmark in 6 
to 11 credits per term. Students were paid at the end of each term if they met the performance 
benchmark. 

This report describes the early findings from Ohio, where the study sample 8 mostly low-income, 
single mothers 8 mirrored that of Louisiana. The program had modest positive impacts on the 
number of credits attempted, number of credits earned, and reduction in loan debt. Unlike in Louisi- 
ana, there was no impact on persistence, though this finding is not surprising given the control 
group® high persistence rate, which was difficult to improve upon. 

Future follow-up reports on the Ohio project and the other demonstration sites will continue to build 
evidence about this promising type of scholarship, which offers a potentially meaningful way to 
increase financial aid to the students with the greatest need and help them succeed in college. 

Gordon L. Berlin 
President 
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Rewarding Progress, Reducing Debt 

Early Results from Ohio’s Performance-Based 
Scholarship Demonstration for Low-Income Parents 




Introduction 



This report presents early results from a rigorous evaluation of a performance-based 
scholarship program that was implemented at three community colleges in Ohio during the 
2008-2009 academic year. Performance-based scholarships are a new and innovative type of 
financial aid for postsecondary education. The scholarships are need-based and are contingent 
upon meeting academic benchmarks after students enroll in the program, and do not place weight 
on past grades. This design is intended to give students an incentive to change their behavior with 
respect to time management and academics, with the goal of improving their educational 
outcomes. It also serves students who may not have had much past academic success, and gives 
them a clean slate. Researchers, institutions of higher education, and policy experts have a 
growing interest in perfomiance-based scholarships as a tool to promote academic success, 
particularly for low-income populations. 

The program in Ohio that is the subject of this report is part of MDRC’s national Per- 
formance-Based Scholarship (PBS) Demonstration, which was launched in 2008 to evaluate 
whether perfonnance-based scholarships are an effective way to improve persistence among 
low-income college students. 1 To the extent possible, they are paid in addition to existing 
financial aid, and typically result in more money for students. 2 The disbursements are based on 
meeting academic benchmarks in the current academic term, regardless of what happened in 
previous terms. 

Students participating in the Ohio study are low-income parents who are eligible for 
Temporary Assistance for Needy Families (TANF). ’ The evaluation uses an experimental 
design — similar to that used in medical trials to test the effects of a drug, for example — to 
measure the effects of the program on academic success and persistence. Students who con- 
sented to participate in the study were randomly assigned to one of two groups: 

• A control group that received the colleges’ standard financial aid package 

• A program group that received the standard financial aid package and was el- 
igible for the perfonnance-based scholarship 



Richburg-Haycs et al. (October 2009a). 

Potential effects on other aid are explained in detail later in this report. 

’The PBS Demonstration in Ohio was not targeted to students requiring developmental education, nor was 
it designed to particularly affect this group; however, they will be analyzed as a subgroup in a future report. 
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The PBS program in Ohio lasted for one academic year and offered students up to 
$1,800 contingent on earning a grade of “C” or better for 12 or more credits, or up to $900 for 
meeting that benchmark for 6 to 1 1 credits. 4 Students who earned fewer than 6 credits did not 
receive any award, regardless of the grades they earned. 

The awards are generally paid directly to students rather than credited to their accounts 
with their college. Students in Ohio received their full performance-based scholarship award 
even if they had outstanding debt to the college in the form of library fines, parking tickets, or 
the like. The direct payment permits students to use the funds for their most pressing needs, 
whether those needs are books, car repairs, child care, or other financial challenges that may 
disrupt their studies. 

The main findings of this study are: 

• The program increased the number of credits attempted. Program group 
students attempted more credits on average than their control group counter- 
parts in the second program term. 

• The program increased full-time enrollment. Program group students 
were 13 percent more likely to enroll full time than control group students in 
the second program term. 

• The program increased the number of credits earned. Program group 
students earned more credits in both the first and second program terms, and 
earned an average of two full credits more than control group students over 
the program year. 

• The program reduced educational debt. As a result of the program, loans 
made up a smaller proportion of total financial aid for program group stu- 
dents than for control group students. On average, program group students 
were awarded $237 fewer loan dollars. 

• The program has had no impact on term-to-term persistence to date. 

This finding was not unexpected, given that the control group students reg- 
istered in the second program term at a high rate that was difficult to im- 
prove upon. 



4 These are unadjusted credits. The use of adjusted and unadjusted credits is described later in this report, in 
Box 2. 
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In addition to describing the program at the Ohio colleges, implementation of the pro- 
gram, and early impact findings, this report provides background infonnation on the national 
PBS Demonstration, its purpose, and its research design. First, however, it lays out the rationale 
for perfomiance-based scholarships and the Ohio context below. 

Theory of Change 

The theory of change underlying the performance-based scholarship is that conditioning 
additional financial aid on certain benchmarks can influence several mediating variables. Some 
sample members may view the scholarship as an inducement to adopt new behaviors. For 
example, some students might respond by seeking tutoring or working at a paid job for fewer 
hours in order to have more time to study. They may also experience a new sense of efficacy 
and competence if they are rewarded for their academic performance in this way. In some cases, 
the scholarship may encourage students to sustain or intensify positive efforts they already 
make, such as completing assigmnents daily, attending class regularly, or finding reliable child 
care so that their class attendance will not be affected by child care needs. 

Improvement in key academic outcomes could occur through these mediating variables. 
They may influence early educational outcomes while the program is operating, such as terrn- 
to-term persistence, academic performance, and the number of credits completed. These early 
outcomes in Ohio are addressed in this report. In turn, the early educational outcomes may lead 
to changes in later educational outcomes, including graduation or transfer to another postsec- 
ondary institution. If the effects on educational outcomes are positive and strong, the perfor- 
mance-based scholarship could in turn lead to improved labor market outcomes in the short 
tenn. Future reports will cover these later outcomes of the Ohio intervention. 

The Ohio Context 

The PBS program in Ohio had a precursor program that had been created in response to 
positive outcomes from an MDRC study of perfonnance-based scholarships in Louisiana. In the 
Louisiana study, which was part of the multisite Opening Doors demonstration, 5 two communi- 
ty colleges implemented a performance-based scholarship program funded by the state using 
TANF surplus funds. The program was designed for low-income parents who were eligible for 
TANF, and predominantly served single mothers. The program operated in Louisiana from 
2004 to 2005 and provided students with scholarships of up to $1,000 for each of two semesters 



5 The Opening Doors demonstration tested several interventions to improve student success at community 
colleges. See Scrivener (forthcoming) for a synthesis of the findings from the various studies. 
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(up to $2,000 per student in total), paid in increments based on each student’s success in 
meeting key benchmarks: 



• $250 upon enrollment (at least half-time, defined as six or more credit hours) 

• $250 after midterms, contingent upon staying enrolled at least half-time and 
earning a “C” average or better 

• $500 upon completion of courses, with a “C” average or better across all 
courses (overall, not per class) 

Performance-based scholarships were distributed in addition to other financial aid pack- 
ages for which students were eligible. The program also provided dedicated counselors who 
monitored students’ performance, verified that the benchmarks were met, and handed out 
scholarship checks to their students. The scholarships were designed to help offset some of the 
financial burdens of attending college, including living expenses, while also rewarding academ- 
ic achievement. 

The early findings showed that this approach resulted in substantial improvements in 
scholarship recipients’ grades, credit accumulation, and semester-to-semester persistence. 6 For 
example, 65 percent of the program group members registered for courses in their second 
semester, compared with 50 percent of those in a control group (for an increase of 30 percent). 
Moreover, these positive trends extended to the third and fourth semesters, when students were 
no longer eligible to receive the scholarships. These results are strong given how difficult it is to 
increase persistence among community college students. 7 

The Louisiana study is one of a handful of studies that is able to measure the effect of 
additional financial aid on academic success. Factors that are associated with financial need, 
such as low-income status, are also associated with a lack of academic success, making it 
difficult to isolate the effect of additional financial aid on student achievement. However, those 
nonexperimental analyses that have been conducted suggest a positive relationship between 
grant aid and persistence. 8 



6 Brock and Richburg-Hayes (2006); Richburg-Hayes et al. (2009b); Barrow , Richburg-Hayes, Rouse, and 
Brock (2009). 

7 Unfoitunately, the devastation inflicted by Hurricane Katrina has made it impossible to determine any 
longer-term effects of the program, such as its impact on transfer and graduation rates. 

s Bettinger (2004); Choy (2002); Dynarski (2003); Leslie and Brinkman (1987). A recent random assign- 
ment study of a combined financial aid and academic support services intervention also suggests a positive 
relationship for a subgroup of students. See Angrist, Lang, and Oreopoulos (2009). 
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The Ohio legislature was impressed by the outcomes from the performance-based 
scholarship program in Louisiana. When Ohio found itself with a TANF surplus of its own, the 
state created a similar program for low-income parents. The TANF Educational Awards Program 
(TEAP) was a performance-based scholarship program designed and funded by two Ohio 
agencies, and implemented statewide in the 2006-2007 academic year. TEAP was originally 
introduced as a one-time, one-year program. Despite a positive response from students, severe 
state budget constraints forced the Ohio legislature to defund TEAP in the following fiscal year. 

Ohio reintroduced TEAP on a limited basis as part of MDRC’s Performance-Based 
Scholarship Demonstration in the 2008-2009 academic year, again using TANF funds. This 
move was a natural fit with the state’s strategic plan, which has as priorities producing more 
college graduates and keeping more college graduates in Ohio. 9 Low-income parents are an 
attractive population for investment because they are more likely to stay in Ohio after 
graduation. 

The PBS program in Ohio was a performance-based scholarship program implemented 
at three community colleges. The Ohio Department of Job and Family Services (ODJFS) 
funded the scholarship using flexible TANF dollars, 10 and the Ohio Board of Regents (OBR) 
administered the program through its division of State Grants and Scholarships. The three Ohio 
program colleges disbursed scholarships to students first and then submitted a reimbursement 
request to the OBR, which in turn submitted a reimbursement request to the ODJFS. The dollars 
then flowed from the ODJFS to the OBR to the colleges. Students participating in the study are 
low-income parents who are eligible for TANF. 

The Ohio Program Colleges 

The Ohio program was implemented at Lorain County Community College, Owens 
Community College, and Sinclair Community College. The institutions cover three of Ohio’s 
four geographic comers. In northeast Ohio, Lorain enrolled 10,769 students in 2007 in the small 
city of Elyria. 11 In the northwestern part of the state, Owens enrolled 18,154 students in the 
Toledo metro area, including a sizable minority from neighboring Michigan. Located in Dayton 
in southwest Ohio, Sinclair enrolled 15,030 students. Additional information on the colleges is 
listed in Table 1 . All three community colleges are recognized as leaders in innovation. Lorain 



9 Fingerhut (2008). 

10 The Ohio Department of Job and Family Services operates the TANF program, among other programs, 
for the state. 

1 'All enrollment numbers are from the U.S. Department of Education, National Center for Education Statis- 
tics, Integrated Postsecondary Education Data System (2007). All three Ohio colleges have seen large enroll- 
ment surges since the 2008 downturn in the economy, and the larger numbers are not reflected in these data. 
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The Performance-Based Scholarship Demonstration 

Table 1 

Selected Characteristics of Participating Colleges 
Lorain County, Owens, and Sinclair Community Colleges 



Characteristic 


Lorain 


Owens 


Sinclair 


Total students (N) 


10,769 


18,154 


15,030 


Enrollment (%) 


Full-time 


40.3 


34.2 


45.7 


Part-time 


59.7 


65.8 


54.3 


Gender (%) 


Male 


35.9 


55.5 


41.8 


Female 


64.1 


44.5 


58.2 


Age a (%) 


Under 25 years 


60.3 


49.4 


52.7 


25-29 years 


13.0 


17.3 


14.5 


30 years and over 


26.7 


33.3 


32.8 


Race/ethnicity (%) 


Hispanic 


6.7 


4.2 


1.6 


White 


80.8 


79.5 


72.5 


Black 


7.6 


11.5 


15.0 


Asian or Pacific Islander 


1.3 


1.0 


1.6 


Other* 3 


3.6 


3.9 


9.3 


Total cost of attendance 13 ($) 


7,450 


11,568 


9,481 


Tuition and fees 


2,400 


3,594 


1,621 


Books and supplies 


1,200 


1,400 


1,038 


Living expenses 


3,850 


6,574 


6,822 


Financial aid. 2007-2008 d 


Total financial aid ($) 


2,050 


3,639 


2,497 


Federal grant aid (%) 


48.2 


31.6 


40.3 


State and local grant aid (%) 


20.1 


14.0 


25.8 


Institutional grant aid (%) 


12.3 


4.0 


5.1 


Student loan aid (%) 


19.4 


50.3 


28.8 


Estimated unmet need ($) 


5,400 


7,929 


6,984 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Integrated 
Postsecondary Education Data System (IPEDS). 



NOTES: Data on undergraduate degree-seeking students are from fall 2007. Missing values are not 
included in individual variable distributions. Distributions may not add to 100 percent because of 
rounding. 

“Data on age are based on the entire undergraduate student population at each school. 

Hncludes nonresident aliens, American Indians, and Alaskan Natives. 

c Data are based on in-district, first-time, full-time undergraduate students living off campus and not 
with their family for the 2007-2008 academic year. 

d Financial aid information is based on undergraduate first-time degree-seeking students. 







offers a unique program called University Partnership, in which students can earn a bachelor’s 
or master’s degree from a partner four-year institution, such as Cleveland State University or 
Youngstown State University, while continuing to attend courses on Lorain’s campus. Owens 
has a strong outreach program that is accessible to its lowest-income students, including a 
community college one-stop center (created to handle multiple administrative functions in one 
place) in downtown Toledo and an active partnership with the Toledo Public Schools system. 
Sinclair participates in Achieving the Dream: Community Colleges Count, an initiative that 
helps community colleges make data-driven institutional changes intended to increase the 
number of at-risk students who succeed, and has several nationally recognized vocational 
programs, including nursing and culinary arts programs. Lorain and Owens operate on the 
semester system, with two semesters per academic year in addition to various summer sessions 
(which are elective). Sinclair Community College operates on the quarter system, with three 
quarters per academic year, plus one elective quarter during the summer. Throughout this 
report, “term” refers to either a semester or a quarter, depending on the college. 

The Ohio Program Model 

The PBS program in Ohio offered students a scholarship for two consecutive semesters 
at Lorain and Owens, or three consecutive quarters at Sinclair. All program students in the Ohio 
study were offered a full-time scholarship of up to $1,800 contingent on earning a grade of “C” 
or better in 12 or more credits, or a part-time scholarship of up to $900 contingent on meeting 
that benchmark in 6 to 1 1 credits, for one academic year. The awards were paid at the end of 
each academic term and were divided evenly among two consecutive semesters or three 
consecutive quarters; thus, the full-time payment was $900 per semester or $600 per quarter, 
and the part-time payment was $450 per semester or $300 per quarter. Students who were 
participating in the program could earn any combination of part-time award, full-time award, or 
no award over the program terms. Students received regular reminders about the scholarship 
throughout the academic term. Notably, the performance benchmarks for the PBS program are 
defined by performance at the course level. Consequently, a student with less than a cumulative 
2.0 grade point average (GPA) could still earn the scholarship, provided grades of “C” or better 
were earned for the required number of credits — that is, it was possible to earn a “C” for 12 
credits and still have a below-“C” average. 12 

In the version of TEAP that had operated in the 2006-2007 academic year, the awards 
had been paid in two increments per academic term. However, the multiple payment structure 



12 For example, if a student took 15 credits, failed one 3-credit course, and got a “C” in each of her remain- 
ing courses, she would still meet the full-time benchmark. 
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resulted in the reduction of some students’ public benefits eligibility . 13 In order to avoid a 
recurrence of this unintended consequence, MDRC collaborated with the ODJFS to redesign the 
scholarship so as to minimize interaction with benefits eligibility. The resulting payment 
structure of one payment per term addressed the issue, and program group students in Ohio 
have not had their benefits decreased as a result of earning the scholarship. 

The Ohio program’s payment structure differed from the Louisiana model, in which 
three payments were made every semester, and the first of those payments was based on 
enrollment rather than on performance. Consequently, while Louisiana students were able to 
earn at least a partial scholarship without meeting a performance benchmark, their counterparts 
in Ohio did not have that opportunity. All students who earned a perfomiance-based scholarship 
in Ohio had to meet a performance benchmark, and MDRC had expected that this condition 
would lead to a smaller proportion of the students in the Ohio study earning the scholarship than 
was the case in Louisiana, and could produce smaller impacts as a result . 14 Students in the study 
could perceive a promised one-time payment at the end of the academic term as a distant 
reward, so the Ohio program design incorporated e-mail reminders and postcards to keep 
students aware of the potential payoff for hard work. These contacts with the students, described 
in detail later in this report, were designed to be upbeat and informative. 

Also unlike the Louisiana model, the Ohio program tests the effects of a basic perfor- 
mance-based scholarship without a counseling component. In Louisiana, the study was not able 
to disentangle the effects of the scholarship and counseling components since both were offered 
to program students. The PBS study in Ohio will reflect the impact of the scholarship alone. In 
the absence of counselors, each college designated staff to track academic performance through 
internal computer systems and to issue checks to students who successfully completed the 
performance requirements at the end of the term. 

The Broader PBS Demonstration 

The Louisiana performance-based scholarship results are impressive, but the study is 
just one test. The positive findings could have resulted from the location of the study in a state 
with few need-based resources or from the target group of low-income parents (who may be 
more mature and capable of utilizing the scholarship), or a combination of those two aspects. 
The goal of the PBS Demonstration is to build more evidence as to whether performance-based 



13 The regularity of the payments caused the scholarship to be counted as income by some students’ case- 
workers. 

14 A program “impact” is the difference between the outcome achieved by the program group and that 
achieved by the control group, providing an effective comparison of what happens when a program is in place 
and what would have happened in its absence. 
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scholarships help at-risk students succeed academically and persist at higher rates than they 
normally would in the absence of such an intervention. The Ohio study will help determine 
whether a comparable amount of scholarship funds targeted to a similar population in a different 
location would have similar outcomes as those in Louisiana. 15 

Currently, eight colleges and one intermediary across six states are participating in 
PBS: 16 Lorain County, Owens, and Sinclair community colleges in Ohio; the Borough of 
Manhattan and Hostos community colleges in New York; the University of New Mexico; Pima 
Community College in Arizona; Hillsborough Community College in Florida; and a state-level 
intermediary in California. While the amount of the scholarship, duration, performance criteria, 
and target group for the scholarship vary across the sites, all programs offer an incentive 
scholarship designed to address the financial needs of low-income students who are not fully 
covered by existing federal and state financial aid programs. Each state’s intervention is being 
evaluated using random assignment. Random assignment is considered the gold standard in 
program evaluation, and is the research methodology preferred by the U.S. Office of Manage- 
ment and Budget for demonstrating program effectiveness. It ensures that the motivation and 
personal characteristics of students in the program and control groups are the same at the 
beginning of each study; hence, any subsequent differences in educational or other outcomes 
can be attributed with a high level of confidence to the PBS Demonstration. In this way, the 
demonstration is testing whether performance-based scholarships are an effective way to 
improve postsecondary academic persistence and success among low-income students in 
different geographic locations, with different amounts of monies, offered for different durations. 



The Performance-Based Scholarship in Ohio 

Ohio’s Financial Aid Context 

The federal Pell Grant is the primary need-based financial aid program for college stu- 
dents in the United States. During the period covered in this report (the 2008-2009 academic 
year), Ohio offered relatively generous state need-based financial aid, and the Ohio College 
Opportunity Grant (OCOG) was the main vehicle for that aid. The maximum Pell Grant was 
$4,731, and the maximum OCOG grant was $2,496. The actual amounts of the need-based 
awards that a student received were affected by the student’s cost of attendance (COA), Ex- 



l5 The $2,000 scholarship in Louisiana eliminated about 19 percent of unmet need, after loans were ap- 
plied, at the larger of the two participating institutions. The scholarship payment in Ohio was calibrated such 
that the maximum scholarship amount of $1,800 represents about 27 percent of the estimated unmet need for 
students after loans were applied. 

16 An intermediary is an existing scholarship provider or administrator that is not institutionally based. 
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pected Family Contribution (EFC), and enrollment status (that is, full time or part time). Box 1 
explains these terms and provides more details about Ohio’s financial aid for students and its 
implications for the participants in the PBS Demonstration in that state. 

Immediately following the period covered in this report, the generosity of Ohio’s need- 
based financial aid declined precipitously. The OCOG program was restructured and received a 
greatly reduced allocation as a consequence of severe state budget constraints in 2009. Among 
the changes to the program was a decision to terminate eligibility for community college 
students. Starting in the 2009-2010 academic year, students at Ohio community colleges, 
including the students in the PBS study, are no longer eligible for OCOG. The State of Ohio is 
considering measures to alleviate the resulting financial burden for this population, but its 
resources are limited. These changes occurred after the impacts described in this report were 
measured, but the tracking of the students in the Ohio study will continue in the context of a 
state with much lower levels of state need-based aid. 

Target Population 

The PBS program in Ohio was funded through state TANF dollars. As a result, the pro- 
gram required that scholarship recipients be low-income parents of minor children. The criteria 
for program eligibility were: 17 

• At least 1 8 years of age 

• U.S. citizen 

• Resident of Ohio 

• Not incarcerated 

• EFC of zero (see Box l) 18 

• Parent (not necessarily custodial) of a minor child 19 

As parents, many students in the PBS study were older than traditional college-age stu- 
dents, and all were treated as independent for the purposes of receiving financial aid. While 
TANF eligibility was a requirement, students did not need to be recipients of TANF cash 
assistance per se. The Ohio program used an EFC of zero as a proxy for TANF eligibility. This 
is interpreted to mean that a student’s family is unable to contribute any funds to the cost of 

17 ln addition to the listed eligibility criteria, MDRC encouraged the colleges to exclude associate’s and 
bachelor’s degree holders, as well as those within six credits of earning either one of those degrees, in order to 
funnel resources to students who were still in need of a first postsecondary credential. This criterion was not 
binding. Additionally, Owens chose to exclude concurrently enrolled high school students. 

1S EFC of zero was approved by the State of Ohio as a proxy for TANF eligibility in this program. 

19 Pregnant students or legal guardians are also eligible. 
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Box 1 



Financial Aid for Students in Ohio and Its Relevance for 
the Performance-Based Scholarship Program 

A student in Ohio applies for both federal and state aid by filling out the U.S. Department of 
Education’s Free Application for Federal Student Aid (FAFSA). The information contained in 
the FAFSA includes self-reported data on income, assets, family size, and other factors that 
contribute to a student’s ability to pay for higher education. 

The student’s institution also determines a cost of attendance (COA) for the student within 
the guidelines set out by the Fl.S. Department of Education. This figure is generally the sum of 
tuition and fees; an allowance for books, supplies, transportation, and personal expenses; and 
an allowance for room and board. The COA also takes into account individual student cir- 
cumstances, such as whether the student lives at home or is independent of his or her parents. 
All three of the colleges in the Ohio PBS program have relatively low COAs compared with 
those of four-year institutions. Two of the colleges have COAs under $10,000 per student per 
year, and the third college’s figure is slightly higher. These figures, as well as selected demo- 
graphic characteristics of the three Ohio colleges, are detailed in Table 1 of this report. 

The FAFSA and COA data are processed by the U.S. Department of Education to produce the 
following figures: 

Expected Family Contribution (EFC). The EFC is the amount of money a student is ex- 
pected to pay out of pocket, or procure in additional loans, to contribute to the COA. It is 
calculated based on students’ income, family size, state of residence, and a number of other 
factors. All other things being equal, a lower EFC is associated with higher levels of need- 
based aid. 

The Ohio program required students to have an EFC of zero, which resulted in a sample with 
higher-than-average levels of need-based grant aid. 

Unmet Need. Unmet need is defined as COA minus the total value of all financial aid, minus 
EFC. Despite low COAs, typical students at the Ohio program colleges have significant unmet 
need. On average, students at these colleges received approximately $2,000 to $3,600 of 
financial aid in the 2007-2008 academic year, leaving an estimated unmet need of approx- 
imately $5,400 to $8,000. 



attendance. Students with an EFC of zero are eligible for the maximum amount of need-based 
financial aid, so the average student in the Ohio PBS study received higher levels of financial 
aid than the average student at their colleges. As a result of having more financial aid and an 
EFC of zero, these students typically had lower umnet need calculations. However, students 
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with an EFC of zero often have financial challenges that are not reflected in their unmet need 
figures. This is particularly the case for independent students and parents. 20 The students in the 
study were often juggling multiple responsibilities, including work and child care responsibili- 
ties, in addition to attending college. 

Fall term of 2008 marked the beginning of the PBS program in Ohio. The total enroll- 
ment goal for the study was 2,400 students across the three colleges, and students were enrolled 
in multiple cohorts over the course of one academic year. 21 The analyses in this report cover the 
fall 2008 cohort only, which accounts for about 60 percent of the total sample. 22 Program 
implementation is described in the following section. 

Implementation of the Performance-Based Scholarship Program in Ohio 

The process for implementing the PBS program at Lorain, Owens, and Sinclair com- 
munity colleges involved multiple phases. Those phases included recruitment and study intake, 
tracking financial aid changes, contacting students throughout the academic term with scholar- 
ship reminders, monitoring student academic progress, and disbursing scholarship money. For 
all those activities, the three colleges designated staff to manage and implement the program. 
The Director and Associate Director of Financial Aid oversaw all aspects of program imple- 
mentation at Lorain and Sinclair, respectively; at Owens, the Vice President of Student Services 
shared responsibility for program implementation with the Associate Director of Financial Aid. 
In addition to the senior staff who managed the programs, all three colleges had a coordinator 
who was responsible for day-to-day operations. Finally, the role of data liaison was of particular 
importance in the Ohio PBS program because timely acquisition of study students’ financial aid 
award information was crucial to tracking changes in financial aid across the program and 
control groups. Owing to a confluence of factors — including the state financial aid situation, 
the low cost of attending a community college, and the characteristics of the target group — the 
Ohio program had a stronger likelihood of changing the composition of students’ financial aid 
packages than did other sites in the PBS Demonstration. 

Each Ohio college tailored its PBS program implementation to its own institutional cul- 
ture. While key research-related aspects of the PBS program were standardized, allowing 



20 Independent students are treated less advantageously by financial aid formulas than are dependent stu- 
dents. Students who are parents are automatically classified as independent. 

2l The study cohorts were fall 2008 (the subject of this report), winter 2009 (Sinclair only), and spring 
2009. 

22 In this study, students are randomly assigned just prior to the start of a term and are then considered to be 
part of that term’s cohort. In other words, a student who was randomly assigned near the start of the fall 2008 
term is a member of the fall 2008 cohort. 
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college-specific customization was important for evaluating the program as it would run outside 
of a research demonstration. The variation in implementation across colleges is expanded upon 
in the following sections. 

Recruitment and Intake 

Lorain and Owens had participated in a previous MDRC study , 23 and thus, had first- 
hand familiarity with the challenges of recruiting students for a random assignment study. All 
three Ohio colleges committed to a strong campaign to reach the PBS target population of low- 
income parents. Eligible students with an EFC of zero and minor children were identified 
through internal financial aid databases at each college and were invited via e-mail or letter to 
an information session on campus. 

Both Lorain and Sinclair used group sessions to present information about the perfor- 
mance-based scholarship to eligible students, whereas Owens opted to hold one-on-one sessions 
with students. Whether the session was group or individual, the colleges covered the same 
topics: the study and random assignment, the opportunity to earn a performance-based scholar- 
ship, and possible consequence to the students’ existing financial aid. The three colleges 
explained potential changes to students’ existing financial aid during the intake session, and 
Owens presented each student’s actual financial aid implications in its one-on-one sessions. 
Although there was a risk that program group students would not earn the award, the Ohio 
colleges recognized that a performance-based scholarship was preferable to loan debt for their 
students . 24 The scholarships were intended to decrease some loan debt in the cases where the 
full scholarship amount could not be awarded in addition to existing financial aid. As a conse- 
quence, depending on their individual financial aid packages, some students did not experience 
the full “dosage” of the program. 

After learning about the limited risks of participating in the study, interested students 
gave their written informed consent to participate. The students then supplied MDRC with 
their contact information and filled out a Baseline Information Form (BIF) giving their 
demographic characteristics at the time of recruitment into the study. The last step of the intake 
process was random assignment into program and control groups. Staff at both Lorain and 
Sinclair believed it was important to let time pass between the informational session and 
notification of group assignments, and that the group environment was inappropriate for 
communicating random assignments. Thus, MDRC randomly assigned the students from 



23 See, for example. Scrivener and Weiss (2009). 

24 A minority of students had insufficient unmet need to accommodate the full performance-based scholar- 
ship. Federal regulations prohibit the awarding of financial aid in excess of unmet need. 
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Lorain and Sinclair in weekly batches, and the schools notified students of their assignment 
approximately one week after their intake session via letters containing standardized language. 
Owens, since it was performing intake in one-on-one sessions, believed that its situation was 
appropriate for immediately communicating random assignments to both study groups. As a 
result, Owens used MDRC’s online random assigmnent module and communicated program 
or control group status to students. The Ohio program’s process of recruitment and random 
assigmnent is represented in Figure 1. 



The Performance-Based Scholarship Demonstration 
Figure 1 

Random Assignment Process Used for the Demonstration 
Lorain County, Owens, and Sinclair Community Colleges 
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The program intervention involved the addition of the performance-based scholarship 
to students’ financial aid, so students did not need to return to campus to complete the process 
of accessing the program. Rather, once students were informed of their assigmnents, the staff at 
Lorain, Owens, and Sinclair added the performance-based scholarship to the program students’ 
financial aid packages. As a result of the hard work and dedication of the staff at the Ohio 
program colleges, the recruitment effort was very successful at all three colleges. 

Reminders to Students 

The Ohio program colleges and MDRC recognized that students could conceivably 
forget about the PBS program over the course of the academic term since the program has a 
single payment at the end of the term. To address this possibility, MDRC worked with Lorain, 
Owens, and Sinclair to craft upbeat e-mail reminders and postcards to send to students through- 
out the academic term. The language was standardized and designed to remind students of the 
value of higher education, the potential to earn the performance-based scholarship, and the 
availability of academic resources on each campus. They also contained information about the 
performance benchmark and scholarship amounts. Students in the program group received six 
e-mail reminders per academic year — three per semester at Lorain and Owens, and two per 
quarter at Sinclair. The timing of the reminders was designed to coincide with crucial points 
during each term, such as students’ midterm and final exams, in order to help them stay focused 
on the performance-based scholarship payment. 

Monitoring and Scholarship Disbursement 

The progress of program group students was checked at the end of the academic term to 
determine whether they were eligible to receive the perfonnance-based scholarship payment. 
Given that this program did not have a counseling component, the scholarship payment staff 
restricted their interaction with students to answering questions if students called either to find 
out when they would receive payments or to leam what other financial aid options were 
available to them in the event that they missed scholarship payments. At the end of the academ- 
ic term, the colleges reviewed students’ transcripts to verify whether they had met the bench- 
mark. The payments were made by check, direct deposit to a checking account, or direct deposit 
to a campus spending card. 

MDRC visited the three Ohio colleges one time each, in the spring of 2009, to confirm 
accurate performance tracking and award disbursement of the perfonnance-based scholarship. 
Lorain, Owens, and Sinclair walked MDRC through their performance monitoring system, their 
disbursement protocol, and financial aid databases in a step-by-step fashion. In addition, MDRC 
reviewed the transcripts of a randomly selected sample of program students to confirm that 
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course-level performance had been accurately assessed, and that the appropriate payments (part 
time, full time, or none) had been made to the students. All three colleges demonstrated a strong 
capacity for the accurate monitoring of student progress and disbursement of the perfonnance- 
based scholarship. 

After the awards were made, program students received a follow-up letter as an addi- 
tional reinforcement to help them keep their future scholarship eligibility in mind. Students who 
met the benchmark and received the payment were congratulated on their hard work and 
encouraged to maintain their level of effort in future academic tenns. Students who failed to 
meet the benchmark were reminded about the scholarship and reassured that, although they had 
not earned the award in this term, they had a remaining tenn or terms of program eligibility 
during which they could earn the scholarship. The fall 2008 cohort students at Lorain and 
Owens had one semester of remaining eligibility; the fall 2008 cohort students at Sinclair had 
two quarters of remaining eligibility. (See Box 2 for an explanation of the differences between 
semester-based and quarter-based schools.) They did not, however, have their eligibility 
extended beyond the program period of one academic year. 

Data Sources 

Several data sources are used in the analyses presented in this report. First, as mentioned earlier, 
students completed a questionnaire on baseline infonnation before being randomly assigned to 
the program group or control group. The BIF covered a range of demographic and other 
background infonnation on students prior to their participation in, and thus any influence by, the 
program. Baseline data are used to describe the sample and assess the success of random 
assignment, and in future reports may be used to make statistical adjustments and classify 
subgroups of sample members. Second, MDRC obtained financial aid data for program and 
control group members before and after random assignment took place. Those data permit an 
examination of how the addition of the performance-based scholarships affected students’ 
financial aid packages. Third, the three colleges provided MDRC with scholarship payment 
records, which are presented in this report to answer the question, “Was there a program?” Last, 
the Ohio Board of Regents (OBR) provided student-level transcript data for the program group 
and control group members in the study. The OBR compiles data supplied by Ohio’s colleges 
and universities and keeps a comprehensive database with data on students, courses, faculty, 
facilities, and finances. The OBR provided data on individual courses taken by each student, 
their credit value, and whether the student earned the credit. Those data are used to provide a 
detailed look at sample members’ performance in college through various measures such as 
enrollment, credits attempted, and credits earned. Data from the OBR also permit tracking 
students who transfer within Ohio postsecondary institutions. Such transfer students will be 
investigated in a future report. 
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Box 2 



Semester-Based Schools Versus Quarter-Based Schools 



The academic year at Lorain County Community College and Owens Community College 
was based on a semester system, while at Sinclair Community College, the academic year 
was based on a quarter system. Semester-based schools such as Lorain and Owens gener- 
ally have two semesters of 1 5 weeks each during the fall and spring, plus optional summer 
sessions of varying lengths. Quarter-based schools like Sinclair usually divide the calen- 
dar year into four quarters, each about 1 0 weeks long, with three quarters (fall, winter, and 
spring) constituting one academic year (and summer is optional). Thus, in an academic 
year, both semester-based schools and quarter-based schools have approximately 30 
weeks of instruction, with two semesters equivalent to three quarters. 

Both semester-based and quarter-based schools use credits to define the number of aca- 
demic hours of instruction for a course, per week. For example, a three-credit course 
would generally have three hours of instruction in a week, regardless of whether it was 
taken at a semester-based or quarter-based school. However, this same three-credit course 
would take 15 weeks to complete at a semester-based school, while taking only 10 weeks 
to complete at a quarter-based school. To adjust for this, the two types of schools general- 
ly require a different number of credits for a degree. Lorain and Owens generally require 
60 to 75 “semester credits” for an associate’s degree, while at Sinclair the range is be- 
tween 90 and 110 “quarter credits.” Semester-based schools use semester credits to define 
part-time and full-time enrollment, and quarter-based schools use quarter credits to define 
part-time and full-time enrollment. 

In order to show these two types of schools equivalently, credits attempted and earned at 
Sinclair have been multiplied by two-thirds in the tables in this report. This adjustment 
equalizes the credits and is consistent with the way in which schools deal with credits 
transferred between quarter-based and semester-based schools, as well as how schools 
convert from one system to the other. Using these semester-equivalent credits (or adjusted 
credits) at Sinclair, the results from these three schools can be pooled. 

Throughout this report, part-time enrollment and credits earned refers to 6 to 11 unad- 
justed credits, and full-time enrollment and credits earned refers to 12 or more unadjusted 
credits. Similarly, a part-time scholarship is awarded if the student earned a “C” or better 
for 6 to 1 1 unadjusted credits, and a full-time scholarship is awarded if the student earned 
a “C” or better for 12 or more unadjusted credits. 
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Characteristics of the Performance-Based Scholarship Sample 

The full sample of students in the Ohio colleges comprises 2,285 students. Of this 
number, 60 percent of students were randomly assigned to the program group, and the rest 
were assigned to the control group. As already noted, this report presents the early findings 
of the first cohort (fall 2008), which consists of nearly 60 percent of the full sample of 
students in the Ohio colleges (1,329 students out of 2,285). Table 2 presents the demo- 
graphic characteristics of the students in the fall 2008 cohort who enrolled in the PBS study 
at all three Ohio colleges. 

The sample represents a nontraditional, college-going population, with the average 
age being around 30 years, and over 60 percent of the sample age 27 or above. Almost half 
graduated from high school or received their General Educational Development (GED) 
certificate more than 10 years prior to enrolling in the study. In addition, the sample is 
overwhelmingly female, with only 10 percent of students being male. Close to 80 percent 
are unmarried, and nearly all of the sample members have at least one child, as required by 
the eligibility criteria. 25 Almost one-third of the sample reported having three or more 
children. Together, these characteristics indicate that the sample is made up primarily of 
single mothers. In fact, more than 70 percent of the sample members are unmarried female 
parents (not shown in the table). 26 

Fewer than 2 percent of the sample members are financially dependent on their par- 
ents, and more than 70 percent of the sample said that someone in their household received 
government benefits such as food stamps, cash assistance or welfare, Section 8 or public 
housing assistance, and so forth. Almost half of the sample members were employed at the 
time they completed the BIF, and of those who were employed, 60 percent were working 
more than 20 hours per week. Thus, many sample members were balancing family duties 
with school and work responsibilities. 

About a third of the students reported that they were the first person in their family 
to attend college, and two-thirds said that their main reason for enrolling in college was to 
obtain an associate’s degree. 



25 Two students in the table are shown as having no children but those students were confirmed to be either 
pregnant or the legal guardian of a minor child at the time of intake and were thus eligible for the scholarship. 

26 This percentage includes students who are unmarried and living with their partner. Female parents who 
are unmarried and not living with their partner account for over 60 percent of the sample (not shown in table). 
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The Performance-Based Scholarship Demonstration 
Table 2 

Selected Characteristics of Sample Members at Baseline: Fall 2008 Cohort 



Lorain County, Owens, and Sinclair Community Colleges 



Characteristic 


Full 

Sample 


Gender (%) 


Male 


10.4 


Female 


89.6 


Age (%) 


18-26 years 


39.7 


27-30 years 


19.7 


3 1 years and over 


40.6 


Average age (years) 


30.1 


Marital status (%) 


Married 


21.1 


Unmarried 


78.9 


Race/ethnicity a (%) 


Hispanic 


8.7 


White 


54.8 


Black 


31.3 


Asian or Pacific Islander 


0.5 


Other 


4.6 


Number of children (%) 


0 


0.2 


1 


40.2 


2 


29.3 


3 or more 


30.4 


Among sample members with children 


Average age of youngest child (years) 


5.0 


Household receiving any government benefits 11 (%) 


70.4 


Financially dependent on parents (%) 


1.7 


Currently employed (%) 


49.8 


Among those currently employed 

Number of hours worked per week in current job (%) 


1-10 


7.4 


11-20 


32.6 


21-30 


30.9 


31-40 


27.2 


More than 40 


1.8 


Average hourly wage at current job ($) 


9.1 



(continued) 
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Table 2 (continued) 



Characteristic 


Full 

Sample 


Highest grade completed (%) 




10 or lower 


11.4 


11 


10.9 


12 


77.7 


Diplomas/degrees earned 0 (%) 




High school diploma 


74.8 


GED certificate 


22.8 


Occupational/technical certificate 


15.5 


Associate's degree or higher 


0.7 


None of the above 


1.5 


Date of high school graduation/GED certificate receipt (%) 




During the past year 


5.2 


Between 1 and 5 years ago 


19.5 


Between 5 and 10 years ago 


28.3 


More than 10 years ago 


47.1 


Main reason for enrolling in college 0 (%) 




To complete a certificate program 


6.6 


To obtain an associate's degree 


67.4 


To transfer to a 4-year college/university 


21.4 


To obtain/update job skills 


4.9 


Other 


2.1 


First person in family to attend college (%) 


30.7 


Highest degree/diploma earned by father d (%) 




Not a high school graduate 


23.4 


High school diploma or GED certificate 


46.4 


Some college or associate's degree 


20.2 


Bachelor's degree or higher 


10.1 


Highest degree/diploma earned by mother 0 (%) 




Not a high school graduate 


17.5 


High school diploma or GED certificate 


42.3 


Some college or associate's degree 


30.5 


Bachelor's degree or higher 


9.7 


Language other than English spoken regularly in home (%) 


4.1 


Sample Size 


1,329 


(continued) 
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Table 2 (continued) 



SOURCE: MDRC calculations using Baseline Information Form (BIF) data. 

NOTES: Characteristics shown in italic type are calculated for a proportion of the full sample. 

Missing values are not included in individual variable distributions. Any characteristic with more than 5 
percent of the sample missing a response is noted in the footnotes. 

Distributions may not add to 100 percent because of rounding. 

a Respondents who said they are Hispanic and chose a race are included only in the Hispanic category. 
Respondents who said they are not Hispanic and chose more than one race are considered multiracial. These 
respondents, combined with those who said they are American Indian /Alaskan Native or another race/ethnicity, 
are included in “Other.” 

b Benefits include unemployment/dislocated worker benefits, Supplemental Security Income (SSI) or 
disability, cash assistance or welfare, food stamps, and Section 8 or public housing. 

c Distributions may not add to 100 percent because categories are not mutually exclusive. 
d Excludes 22 percent of the sample who either did not know or declined to answer this question. 
e Excludes 9 percent of the sample who either did not know or declined to answer this question. 



Appendix Table A.l shows the same demographic characteristics as those reported in 
Table 2 for the full sample, the program group, and the control group of the fall 2008 cohort. An 
asterisk in the far-right column of the table indicates that the percentage of program group 
members with that characteristic is significantly different from the percentage of control group 
members. This means that there is only a small probability that the observed difference occurred 
by chance. There are a few differences between the two research groups, but no more than 
would be expected to occur by chance. 27 



Effect of the Performance-Based Scholarship Program on the 
Financial Aid Package 

A student’s financial aid package contains all the various forms of aid that a student is 
scheduled to receive in a specific term or academic year. Its composition relies on numerous 
moving parts such as the characteristics of the U.S. Department of Education’s Free Application 
for Federal Student Aid, or FAFSA (described earlier, in Box 1), student preferences, and 
institutional rules. The resulting financial aid package is extremely complex and may change 
many times before a disbursement is made to the student. When additional forms of aid are 
added to or removed from the total aid package (as was the case with the performance-based 
scholarship), the financial aid is said to be “repackaged.” 



27 In addition, an omnibus test was conducted to assess whether overall systematic differences in baseline 
characteristics were observed between the two research groups. The model’s likelihood ratio test yielded a p- 
value of 0.83. Convention suggests that this probability of differences occurring by chance is large enough that 
these differences can be ignored in the analysis. 
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As explained earlier in Box 1 , unmet need for the Ohio program population is the cost 
of attendance minus the individual student’s financial aid package, minus the student’s Ex- 
pected Family Contribution. In situations where students had umnet need that was lower than 
the amount of the performance-based scholarship, the addition of the scholarship had the 
potential to reduce other forms of aid, usually loans. This possibility is partly a result of the 
nature of the Ohio community colleges in the study, which have lower costs of attendance than 
the average Ohio community college. In addition, the eligibility criteria for the scholarship 
required students to have an EFC of zero, which meant that these students were eligible for the 
maximum amounts of federal and state need-based financial aid. Students with high levels of 
umnet need would have ample room in their financial aid package for the scholarship, and could 
add it without affecting their other aid dollars. 

Though the PBS intervention was not geared specifically to reduce loans, the colleges 
and MDRC agreed that loans should be reduced first in cases where students had low levels of 
umnet need, before any other fomis of aid were changed. There was a general consensus that 
while reducing loans may be troublesome for some students in the short tenn, the reduction in 
educational debt would be better for students in the long run. Recent data from the U.S. Depart- 
ment of Education’s National Postsecondary Student Aid Study reveal that of the students who 
earned an associate’s degree in the 2007-2008 academic year, 48 percent graduated with some 
student debt. 28 For students who are able to successfully enter the workforce, average debt levels 
are manageable, but other students may find themselves with burdensome repayment obligations 
that are difficult to meet. Thus, the Ohio program colleges and MDRC worked to ensure that 
additional scholarship dollars would reduce the amount of loans in student packages first, before 
affecting other aid, and that students understood their ultimate financial aid package. 

While most students were pleased with the reduction in their loan debt, some students 
struggled with the one-time payment of the scholarship at the end of the tenn. The structure was 
difficult for students with immediate financial need who could not afford to wait until the end of 
the academic tenn, and was particularly challenging in the first tenn of the scholarship. Because 
of the sensitivities around student preferences for loans instead of grants in these specific 
circumstances, students were given the choice to opt out of the scholarship for one or more 
tenns. In such cases, the scholarship was not added to their financial aid package during any opt- 
out tenn. However, they could opt back into the scholarship program and would maintain their 



28 Steele and Baum (2009). 
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award eligibility for any remaining terms of the scholarship. In the fall 2008 cohort, only 14 
students selected this option for one or more tenns (around 1 percent). 29 

In order to track and monitor changes to financial aid packages across program and con- 
trol group students, financial aid “snapshots” were collected, which detailed students’ aid 
packages for the term at particular points in time. By collecting snapshots of students’ financial 
aid packages before and after the scholarship was added, the number of students affected by the 
repackaging of aid and the net effect of this repackaging could be determined. In addition, the 
snapshots would affirm whether or not the scholarship reduced student loan debt as originally 
intended. These snapshots indicate the financial aid that was awarded to the student and the 
dollar amount for which the student was eligible. They are not necessarily the amounts that 
were disbursed to students, as these payments can happen at various points during the term. 

Table 3 shows the results of this analysis (reflecting financial aid that was awarded but 
not necessarily received). 30 The first column shows the results for the program group, the 
second column shows the results for the control group, and the third column shows the differ- 
ence between the two groups. An asterisk indicates that the outcome for program group mem- 
bers is significantly different from the outcome for control group members. The far-right 
column shows the standard error of the impact estimate (a measure of uncertainty or variability 
around the impact estimate). 

The first panel in Table 3 (“Initial snapshot”) provides a baseline look at study students’ 
financial aid packages before the perfonnance-based scholarship was included. It shows the 
average dollar amount awarded for the various categories of aid (Pell Grants, other grants, 
subsidized loans, unsubsidized loans, and Federal Work-Study), across all students who have a 
snapshot. On average, study students were awarded more than $4,000 in financial aid, with a 
large proportion of this aid coming from federal Pell Grants, but also from other grants and 
loans. As expected, there are no major differences between the program group and the control 
group at baseline. 

The second panel in the table (“Midterm snapshot”) shows similar information after the 
scholarship was added to the package. In tenns of total financial aid dollars, program group 
members were awarded, on average, $340 more than control group members. While program 



29 An additional 35 students from the fall 2008 cohort chose to withdraw from the study completely. Those 
students are not included in any of the analyses completed for this report. The final report will provide more 
detail about recruitment and withdrawals. 

30 This analysis excludes 115 students, representing 9 percent of the overall sample, for whom this infor- 
mation was not available (29 students who did not have an initial or midterm snapshot, and 86 students for 
whom financial aid was repackaged before an initial snapshot could be collected). 
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The Performance-Based Scholarship Demonstration 
Table 3 



Impacts on Financial Aid Awarded During the First Program Term: Fall 2008 Cohort 
Lorain County, Owens, and Sinclair Community Colleges 



Outcome 


Program 

Group 


Control 

Group 


Difference 

(Impact) 


Standard 

Error 


Initial snapshot IS) 


Total financial assistance awarded 


4,035 


4,060 


-25 


94.2 


Federal Pell grant 


1,790 


1,815 


-26 


33.2 


Performance-based scholarship 


0 


0 


0 


0.0 


Other grants 3 


712 


690 


21 


25.0 


Subsidized loans 


796 


812 


-16 


40.2 


Unsubsidized loans 


702 


694 


8 


49.2 


Federal Work Study 


36 


50 


-13 


17.5 


Midterm snapshot IS) 


Total financial assistance awarded 


4,504 


4,164 


340 *** 


91.6 


Federal Pell grant 


1,699 


1,754 


-55 


33.5 


Performance-based scholarship 


676 


0 


676 *** 


oo 
o o 


Other grants 3 


654 


659 


-5 


26.5 


Subsidized loans 


779 


901 


-122 *** 


38.6 


Unsubsidized loans 


650 


765 


-115 ** 


47.0 


Federal Work Study 


45 


84 


-39 * 


21.7 


Loan displacement 1%) 

Complete removal of subsidized loans 


from initial to midterm 


2.2 


0.6 


1 7 ** 


0.7 


Complete removal of unsubsidized loans 


from initial to midterm 


9.4 


0.5 


g 9 *** 


1.3 


Loans decreased by 10 percent or more 


of the initial total aid awarded 


25.9 


1.1 


24.8 *** 


1.9 


Sample size (total = 1,214) 


720 


494 







SOURCES: MDRC calculations from financial aid data provided by Lorain County, Owens, and Sinclair 
Community Colleges. This excludes students who did not have an initial or midterm snapshot (29 students: 14 
program group, 15 control group). It further excludes Lorain students whose financial aid was repackaged 
before the initial snapshot could be collected (86 students: 48 program group, 38 control group). 

NOTES: Distributions may not add to 100 percent because of rounding. 

A two-tailed t-test was applied to differences between the research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by campus. 

Reported figures are based on amounts awarded to students, not actual receipt. 

a This measure includes all grants and scholarships excluding the Pell Grant and the full-time and part-time 
performance-based scholarships. The Ohio College Opportunity Grant is thus included in this figure. 
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group students were awarded, on average, $676 more in performance-based scholarship dollars, 
they were also awarded $122 less in subsidized loans, and $115 less in unsubsidized loans. 
These results are all statistically significant, and indicate that the scholarship is replacing some 
student loans. Loans make up a lower proportion of total financial aid for program group 
students than for control group students. While this finding was expected, the fact remains that 
though students were awarded $676 more in scholarship dollars, their total aid increased by 
only $340. The effects of the program are somewhat diluted in that some of the benefit of the 
program occurred through a reduction in loans rather than a full increase in aid. Therefore, the 
findings presented here should be viewed with that result in mind. 

The third panel depicts some interesting measures on loan displacement. The first two 
rows show the proportion of students who had a subsidized or unsubsidized loan in the initial 
snapshot, but not the midterm snapshot. Not all loans are considered equal. In general, subsidized 
loans are more advantageous to students because the government pays the interest on the loan 
while the student is in school. Thus, the displacement of unsubsidized loans is preferred over 
subsidized loans, and schools tend to reduce unsubsidized loans in students’ aid packages before 
touching subsidized loans. As expected, there were very few students who had a subsidized loan 
in the initial snapshot but then lost it completely, in either the program group or the control 
group. However, 9.4 percent of program group members had an unsubsidized loan in the initial 
snapshot and no unsubsidized loan in the midterm snapshot, versus 0.5 percent of control group 
members. This difference is statistically significant. The last measure shows the proportion of 
students who had their loans decreased by 10 percent or more of their total initial aid awarded 
(or by around $400). More than 25 percent of program group students had a loan reduction of 
10 percent or more of their total initial aid package, versus only 1 percent of control group 
students. This 24.8 percentage point impact is statistically significant. These results are in line 
with the desired outcome of the program. 

Students who elected to keep the scholarship in their package and reduce their loans, 
but failed to meet the benchmark for the scholarship at the end of the term, were in a somewhat 
precarious position. They did not take as many loan dollars at the beginning of the term, and, 
because their grades were poor, they did not get their anticipated aid. Of the students who did 
not earn the scholarship, around one-third had reduced their loans from the initial to midterm 
snapshot, possibly to make room for the scholarship in their package (around 9 percent of the 
program group with financial aid snapshots available). While some of these students may have 
been hindered by having fewer aid dollars ultimately awarded to them, the positive news is that 
they are not left with additional loan debt arising from courses that they may have failed or from 
which they withdrew. 



27 




Scholarship Receipt 

Students in the program group at Lorain and Owens were eligible to receive a full-time 
performance-based scholarship of $900 per semester (or a part-time scholarship of $450 per 
semester) for two consecutive semesters following random assignment. Students in the program 
group at Sinclair were eligible to receive a full-time scholarship of $600 per quarter (or a part- 
time scholarship of $300 per quarter) for three consecutive quarters following random assign- 
ment. Each scholarship was distributed in one payment at the end of the term upon verifying 
that the student met the performance benchmark. At all schools, the maximum total scholarship 
over the year was $1,800. As explained earlier in Box 2, the part-time and full-time awards 
were based on unadjusted credits. That is, the receipt of a full-time scholarship at all three 
schools means that the student earned a “C” or better in courses totaling 12 or more unadjusted 
credits, and the receipt of a part-time scholarship indicates a “C” or better in courses totaling 6 
to 1 1 unadjusted credits. 

The proportion of students in the program group who earned the scholarship measures 
the level of participation in the program. Table 4 presents the scholarship receipt rates for the 
program year of the first cohort. During the first program term, almost three-fourths of program 
group students received a full-time or part-time scholarship, with 33 percent meeting the full- 
time benchmark and 4 1 percent meeting the part-time benchmark. 

The second panel of Table 4 shows that about 60 percent of program group members 
received a scholarship payment in their second program term. Here, the split between part-time 
and full-time scholarships is about 30 percent each. 

While the first two panels of Table 4 show the first two terms for all three colleges in 
the study, the third panel reflects Sinclair’s program group students only, who were eligible for 
the scholarship over three quarters instead of two semesters. In this final quarter at Sinclair, 
almost 60 percent of program group students received a scholarship, with 26 percent receiving a 
full-time award and 3 1 percent receiving a part-time award. 

The final panel of Table 4 summarizes the entire program year for all three schools. 
Over 80 percent of students received a scholarship in at least one of the program terms, indicat- 
ing that some students missed the performance benchmark in the first program term, but were 
able to meet the criteria in the second program temi. It also reflects that a group of students 
never registered, dropped out, or were unable to earn the perfonnance benchmark in any term. 
In addition, just over half of the students received a scholarship in every term in which they 
were eligible, while 17 percent received the full-time award in all program terms. The average 
scholarship amount received over the program year was $868, or approximately 48 percent of 
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The Performance-Based Scholarship Demonstration 
Table 4 



Scholarship Receipt Among the Program Group Members: Fall 2008 Cohort 
Lorain County, Owens, and Sinclair Community Colleges 

Program 

Outcome Group 

First program term (%) 

Received a scholarship payment 73.5 

Received full-time scholarship 32.9 

Received part-time scholarship 40.7 

Second program term (%) 

Received a scholarship payment 60.6 

Received full-time scholarship 29.8 

Received part-time scholarship 30.8 

Sample size 782 

Third program term (Sinclair only ) 3 (%) 

Received a scholarship payment 57.3 

Received full-time scholarship 26.3 

Received part-time scholarship 31.0 

Sample size 255 

Summary 

Received any scholarship payment (%) 80.9 

Received scholarship payments in all payment tenns (%) 51.0 

Received full-time scholarship in all tenns 16.5 

Average total scholarship amount received among all program group members ($) 868 

Average total scholarship among recipients 1,072 

Sample size 782 

SOURCE: MDRC calculations from fall 2008 scholarship payment data provided by Lorain County, Owens, and 
Sinclair Community Colleges. 

NOTES: Items shown in italic type are calculated for a proportion of the full sample. 

Sample members receive a part-time payment at the end of the tenn for completing 6 to 1 1 credits with a 
grade of "C" or higher, or a full-time payment at the end of the term for completing 12 or more credits with a 
grade of "C" or higher. 

a The third program term applies only to Sinclair because the scholarship was available over three quarters 
(versus two semesters at Lorain and Owens). 
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the maximum total scholarship available, assuming a full-time award in each term. 31 Among 
those students who received any scholarship payment, the average scholarship amount received 
over the program year was $1,072. 



An Early Look at Program Impacts 

As discussed earlier, a key goal of evaluating performance-based scholarships is to test 
whether such scholarships increase students’ academic success in college. During the program 
year, it is expected that the scholarship may induce the program group to persist at higher rates, 
attempt more credits, and earn more credits. This section focuses on the impacts of the Ohio 
program on those educational outcomes. 32 Note that for measures such as the number of credits 
attempted and earned, the credits have been adjusted as described earlier in Box 2. The credits 
shown for these measures are semester-equivalent, or adjusted credits, and have been multiplied 
by two-thirds for Sinclair students. Measures such as full-time and part-time enrollment, earned 
12 or more credits, and earned 6 to 1 1 credits are based on unadjusted credits. 

First Program Term 

The top panel of Table 5 shows the results for the first program term. Nearly all the stu- 
dents in the program group and control group registered for at least one course, and they 
registered full time or part time at about the same rate. Similarly, there was no detectable 
difference between program group and control group members in the number of credits at- 
tempted. Given that the point of random assignment generally occurred after students had 
already decided to matriculate, these results are not surprising. 

However, students in the program group earned almost one credit more, on average, 
than students in the control group. Given that students in the program group and the control 
group registered for courses at similar rates, and that there is an impact on the number of credits 
earned, the scholarship appears to be encouraging students to meet the perfomiance benchmark. 
Students in the program group were also 7.0 percentage points more likely to earn 12 or more 
credits within the term than students in the control group, which represents an increase of 24 
percent. Lastly, students in the program group were 12.5 percentage points more likely to pass 
all of the courses they attempted, indicating that the scholarship may have encouraged some 

3 1 This amount is the average amount received over the program year , whereas the amount cited in the 
previous section ($676) was the average amount awarded over the first program term. 

32 No analysis of GPA is included in this report. However, since the scholarship benchmark was based on 
the number of credits earned with a grade of “C” or better, the measure of credits earned is a better proxy for 
whether students met the benchmark. Credits earned include “D” grades, which do not count toward the 
scholarship criteria. An analysis of GPA will be completed in a future report. 
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The Performance-Based Scholarship Demonstration 

Table 5 

Educational Outcomes for the Program Year: Fall 2008 Cohort 



Lorain County, Owens, and Sinclair Community Colleges 



Outcome 


Program 

Group 


Control 

Group 


Difference 


Standard 

Error 


First nroeram term 


Registered for any courses (%) 


97.4 


97.1 


0.3 


0.9 


Enrolled full time 3 


61.4 


64.3 


- 2.8 


2.6 


Enrolled part time 3 


34.8 


31.0 


3.8 


2.6 


Number of credits attempted 13 


9.6 


9.7 


0.0 


0.2 


Regular credits 


8.2 


8.1 


0.1 


0.2 


Developmental credits 


1.4 


1.6 


- 0.1 


0.1 


Number of credits earned 13 


7.4 


6.7 


Q 7 *** 


0.2 


Regular credits 


6.5 


5.9 


0.6 *** 


0.2 


Developmental credits 


0.9 


0.9 


0.1 


0.1 


Earned 12 or more credits 3 (%) 


35.8 


28.9 


7 Q *** 


2.6 


Earned 6 to 1 1 credits 3 (%) 


41.1 


40.7 


0.4 


2.7 


Passed all courses (%) 


57.9 


45.4 


12 5 *** 


2.8 


Second nroeram term 


Registered for any courses (%) 


84.2 


82.5 


1.7 


2.1 


Enrolled full time 3 


55.4 


49.1 


6.3 ** 


2.7 


Enrolled part time 3 


27.5 


29.0 


-1.5 


2.4 


Number of credits attempted 13 


8.5 


8.0 


0.6 ** 


0.3 


Regular credits 


7.7 


7.0 


Q 7 *** 


0.2 


Developmental credits 


0.8 


0.9 


- 0.1 


0.1 


Number of credits earned 13 


6.4 


5.5 


| Q *** 


0.3 


Regular credits 


6.0 


5.0 


| Q *** 


0.3 


Developmental credits 


0.5 


0.5 


0.0 


0.1 


Earned 12 or more credits 3 (%) 


33.1 


20.4 


^2 7 *** 


2.5 


Earned 6 to 1 1 credits 3 (%) 


32.1 


37.0 


-4.9 * 


2.6 


Passed all courses (%) 


46.7 


39.1 


7 ^ *** 


2.8 


Sample size (total = 1,329) 


782 


547 







(continued) 
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Table 5 (continued) 



Outcome 


Program 

Group 


Control 

Group 


Difference 


Standard 

Error 


Third nrosram term (Sinclair onlv) c 










Registered for any courses (%) 


75.7 


73.8 


1.9 


4.2 


Enrolled full time 3 


49.0 


50.3 


-1.2 


4.8 


Enrolled part time 3 


24.3 


19.8 


4.5 


4.0 


Number of credits attempted 13 


5.6 


5.5 


0.1 


0.4 


Regular credits 


5.3 


5.2 


0.1 


0.4 


Developmental credits 


0.3 


0.3 


0.0 


0.1 


Number of credits earned 13 


4.5 


3.6 


Q9** 


0.3 


Regular credits 


4.4 


3.5 


Q 9 *** 


0.3 


Developmental credits 


0.1 


0.1 


0.0 


0.1 


Earned 12 or more credits 3 (%) 


28.2 


24.1 


4.2 


4.3 


Earned 6 to 1 1 credits 3 (%) 


32.5 


23.5 


9Q** 


4.3 


Passed all courses (%) 


45.9 


33.2 


^2 7 *** 


4.7 


Sample size (total = 442) 


255 


187 







SOURCE: MDRC calculations from Ohio Board of Regents transcript data for the fall 2008 cohort. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels 
are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by campus. 

“These data are based on unadjusted credits. “Full time” is defined as a total of 12 or more semester 
or quarter credits during the term — that is, 12 or more unadjusted credits. “Part time” is defined as 6 to 
1 1 semester or quarter credits during the term — that is, 6 to 1 1 unadjusted credits. 

b These data are based on semester-equivalent or adjusted credits. 

“The third program term applies only to Sinclair because the scholarship was available over three 
quarters (versus two semesters at Lorain and Owens). The sample size here is 442 students, versus the full 
fall 2008 cohort sample size of 1,329. 

program group students to remain in their classes rather than withdraw, or motivated them to 
perform better in their initial course load. All of these differences are statistically significant, 
meaning that the differences are not likely due to chance. 

As noted earlier, the PBS Demonstration in Ohio was not targeted to students requiring 
developmental education, nor was it designed to particularly affect this group. Nonetheless, 
over 35 percent of the fall 2008 cohort took at least one developmental education class in the 
first program year (not shown in tables). Students requiring developmental education will be a 
subgroup of interest in a future report, where they will be investigated more thoroughly. 
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Second Program Term 

The second panel of Table 5 shows the results for the second program term. When con- 
sidering that the second term registration rates were 84 percent for the program group and 83 
percent for the control group, it is clear that there was already a high rate of persistence in the 
second term for this cohort. There was not a large problem with retention and the program had 
little impact (and little room for impact) on this measure. However, this outcome will be 
examined closely in future reports. 

In the second program term, program group students were 6.3 percentage points more 
likely to enroll full time than control group students (an increase of 1 3 percent), and attempted 
0.6 more credit on average than control group students. These differences are statistically 
significant. Since the first term’s scholarship was paid at the end of the temi, it is reasonable to 
expect that impacts in attendance status would occur in the second term, not the first. The 
impact on full-time enrollment in the second program term may suggest that the scholarship that 
students received from the first term helped make a full-time course load more affordable for 
students, or that students were motivated by the higher full-time award amount. It is also 
possible that the scholarship enabled working students who would have otherwise attended part 
time to reduce their hours at a job, or enabled some parents to pay for child care while they 
attended more classes. 33 

In the second program term, students in the program group earned one full credit more, 
on average, than students in the control group. Given that students in the program group 
attempted 0.6 credit more on average, but earned 1 credit more, the impact on credits earned is 
not solely due to program group students attempting more credits. This finding is further 
substantiated when examining the proportion of students who passed all their courses. Program 
group students were 7.5 percentage points more likely to pass all their courses. Lastly, students 
in the program group were 12.7 percentage points more likely to earn 12 or more credits than 
students in the control group, an increase of 62 percent. These differences are all statistically 
significant. Thus, even though registration rates were comparable between the two groups, 
program group members attempted more credits in the second program term and ultimately 
earned more credits as well. 

Third Program Term (Sinclair Only) 

The third panel in Table 5 shows the results for the third program term for Sinclair 
Community College. The sample size here is 442 students, or the number of students in the fall 



33 Survey results in a future report will help shed light on which of these factors contributed most to the 
findings. 
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2008 cohort at Sinclair, rather than the full fall 2008 cohort of 1,329 students at all three 
colleges. Thus, the power to detect impacts has been weakened when looking at this Sinclair 
term in isolation. Registration rates drop to about 75 percent overall, with still no detectable 
difference between program and control groups. There is also no impact on full-time or part- 
time enrollment, or on the number of credits attempted. However, there is a statistically signifi- 
cant impact of almost one full credit earned, and students in the program group were 12.7 
percentage points more likely to pass all their courses. 

The Full Program Year 

Table 6 shows the outcomes of interest over the entire program year. For Lorain and 
Owens, the program year for the fall 2008 cohort comprises the fall 2008 and spring 2009 
semesters. For Sinclair, the program year for the fall 2008 cohort comprises the fall 2008, 
winter 2009, and spring 2009 quarters. 

Since almost every student in both the program and control groups had registered for 
the first program term, the measure of registration in any program term consists of almost the 
entire fall 2008 cohort. There are no detectable differences in registration or credits attempted. 

However, program group students earned approximately two credits more than con- 
trol group students over the program year, a 14 percent increase in credits earned. This 
difference is statistically significant. In addition, program group members were 6.6 percentage 
points more likely to earn 24 or more credits than control group members, an increase of 43 
percent. This measure is a proxy for full-time attendance and credit completion during some 
or all of the program terms. Numerous studies have documented that part-time attendance is 
one of the critical risk factors that account for students’ dropping out of college or not earning 
a college degree or credential. 34 It is encouraging that the PBS program is increasing the 
proportion of full-time students who are on track to earn a degree, especially given this 
nontraditional study population. 



Conclusions 

A look at the first program year of the first cohort of the PBS demonstration in Ohio 
suggests that the program encouraged students to earn more credits in all of the program terms, 
and to attempt more credits in the second program term. The program did not have any impact 
on the already high second-term persistence rate, but students in the program group earned 
approximately two full credits more over the year than students in the control group. At the 



34 Provasnik and Planty (2008); Horn and Berger (2005); Hoachlander, Sikora, and Horn (2003). 
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The Performance-Based Scholarship Demonstration 
Table 6 

Cumulative Academic Outcomes for the Program Year: Fall 2008 Cohort 



Lorain County, Owens, and Sinclair Community Colleges 



Outcome 


Program 

Group 


Control 

Group 


Difference 


Standard 

Error 


Registered for any courses (%) 


98.3 


97.8 


0.5 


0.8 


Number of credits attempted 3 


20.0 


19.5 


0.5 


0.4 


Regular credits 


17.7 


16.9 


0.8 * 


0.4 


Developmental credits 


2.4 


2.6 


-0.2 


0.2 


Number of credits earned 3 


15.3 


13.4 


2 Q *** 


0.5 


Regular credits 


13.9 


12.0 


1 9 *** 


0.5 


Developmental credits 


1.4 


1.4 


0.1 


0.2 


Earned 24 or more credits 3 (%) 


22.0 


15.4 




2.2 


Earned 12 to 23 credits 3 (%) 


45.0 


45.0 


-0.1 


2.8 


Sample size (total = 1,329) 


782 


547 







SOURCE: MDRC calculations from Ohio Board of Regents transcript data for the fall 2008 cohort. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by campus. 

The program year is considered the fall 2008 and spring 2009 semesters for semester-based schools, and the 
fall 2008, winter 2009 and spring 2009 quarters for quarter-based schools. 
a These are based on semester-equivalent or adjusted credits. 



institutions in this study, this increase in credits can sometimes account for one full course that 
counts toward a student’s degree requirements. This impact seems like a large one, but whether 
it is enough to produce higher graduation rates in the absence of retention impacts remains to be 
seen. An increase of about two credits over the year may be an indication that these students 
will have a shorter time to degree completion, but without a direct effect on persistence or 
graduation rates, these additional credits are not likely to cause major shifts in employment or 
earnings. Using longitudinal survey data, Kane and Rouse concluded that the average two-year 
and four-year college student earns roughly 5 percent more than high school graduates for every 
30 credits completed. 35 A more recent paper by Jacobson and Mokher finds that median 
earnings were about 27 percent greater for students with a certificate (and 8 percent greater for 



35 Kane and Rouse (1993). This analysis uses data from the National Longitudinal Study of the High 
School Class of 1972. 
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students with an associate’s degree) relative to students who left college without a credential, 
but this analysis was specific to the state of Florida. 36 

As discussed earlier, the PBS demonstration in Ohio tests a population that is very simi- 
lar to the one studied in the Opening Doors demonstration in Louisiana. While Opening Doors 
Louisiana had three payment points throughout each semester and a counseling component, the 
Ohio intervention had a single payment point at the end of the term without counseling. The 
Louisiana program also had a single scholarship amount regardless of part-time or full-time 
status, while Ohio differentiated between part-time and full-time students. Thus, while the 
populations were similar, the same impacts were not necessarily expected. Early results from 
Opening Doors Louisiana showed impacts on credits attempted and earned in the first and 
second program semesters, similar to the impacts seen in the PBS program in Ohio. In the first 
program semester in Louisiana, program group students attempted 0.6 credit more and earned 
1.2 credits more than control group students. In the second semester, program group students 
attempted 1.2 more credits and earned 1.1 more credits than control group students. However, 
part of the second semester impacts in Louisiana were driven by the 15.0 percentage point 
difference in the second semester registration rates. This same impact is not observed in Ohio, 
as both program and control group members had a high rate of persistence in the second term. 
This high persistence rate could be a result of situations that are specific to these Ohio colleges, 
or may more broadly be a result of the economic downturn that began in 2008. Also, the second 
term retention for this Ohio cohort is attributed to the winter and spring terms following the fall 
term, when retention rates are typically higher, whereas the Louisiana impact was pooled over 
all of its cohorts. It is possible that future cohorts and terms in the Ohio program may show 
impacts on retention from the spring to fall terms. Additionally, Louisiana had multiple pay- 
ment points and students received part of their total award for simply enrolling at the college at 
least part time. In Ohio, all students had to meet the performance benchmark to earn the award, 
and there were no partial payments based on enrollment. Thus, MDRC had expected that a 
smaller proportion of program group students in Ohio would earn the scholarship compared 
with the students in Louisiana, and that the Ohio program could produce smaller impacts as a 
result. Lastly, it is possible that the counseling aspect of the Louisiana study played a role in 
helping to explain some of the different impacts that were observed in the two programs. 

The findings in this early report are modest but encouraging. If the program can show 
lasting effects in the absence of the scholarship, and impacts in persistence are seen in the 
postprogram terms, such results could lead to higher graduation rates and translate into higher 



36 Jacobson and Mokher (2009). This analysis was completed based on comprehensive data from the state 
of Florida belonging to a cohort of 144,545 students in the ninth grade in 1996. 
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earnings. This is an early report on the first cohort, and these educational outcomes will be 
followed closely in the coming terms. 



Next Steps 

A research report on longer-term findings will be released in fall 2011. The report will 
contain findings from an in-depth implementation study, a qualitative study, and the impact 
study. The implementation study will describe in greater detail how the performance-based 
scholarship program operates, identify promising recruitment strategies and other best practices, 
and analyze whether the program operated as intended. The qualitative study will capture the 
experiences and insights of students and administrators through focus groups and interviews. 
The impact study will allow for a causal interpretation of the effect of performance-based 
scholarships on educational outcomes from administrative records (including student retention, 
credit accumulation, and graduation and transfer rates) and from a student survey (such as time 
spent studying, motivation, and use of scholarship monies). 
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Appendix A 

Demographic Characteristics for the Full Sample, 
the Program Group, and the Control Group 




The Performance-Based Scholarship Demonstration 
Appendix Table A.l 

Selected Characteristics of Sample Members at Baseline: Fall 2008 Cohort 
Lorain County, Owens, and Sinclair Community Colleges 

Full Program Control 

Characteristic Sample Group Group 

Gender (%) 



Male 


10.4 


9.5 


11.7 


Female 


89.6 


90.5 


88.3 


Age (%) 


18-26 years 


39.7 


39.8 


39.7 


27-30 years 


19.7 


20.0 


19.4 


3 1 years and over 


40.6 


40.3 


40.9 


Average age (years) 


30.1 


30.1 


30.0 


Marital status (%) 


Married 


21.1 


21.9 


20.0 


Unmarried 


78.9 


78.1 


80.0 


Race/ethnicity 3 (%) 


Flispanic 


8.7 


8.4 


9.2 


White 


54.8 


56.0 


53.1 


Black 


31.3 


30.0 


33.3 


Asian or Pacific Islander 


0.5 


0.5 


0.6 


Other 


4.6 


5.1 


3.9 


Number of children (%) 


0 


0.2 


0.3 


0.0 


1 


40.2 


39.1 


41.7 


2 


29.3 


30.4 


27.8 


3 or more 


30.4 


30.2 


30.5 


Among sample members with children 


Average age of youngest child (years) 


5.0 


5.0 


5.0 


Household receiving any government benefits' 3 (%) 


70.4 


68.1 


73.6 ** 


Financially dependent on parents (%) 


1.7 


1.7 


1.7 


Currently employed (%) 


49.8 


49.2 


50.7 


Among those currently employed 
Number of hours worked per week in current job (%) 


1-10 


7.4 


8.0 


6.6 


11-20 


32.6 


30.8 


35.1 


21-30 


30.9 


32.4 


28.9 


31-40 


27.2 


26.2 


28.7 


More than 40 


1.8 


2.7 


0.7 


Average hourly wage at current job ($) 


9.1 


9.2 


9.0 



(continued) 
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Appendix Table A.l (continued) 



Characteristic 


Full 

Sample 


Program 

Group 


Control 

Group 


Highest grade completed (%) 


10 or lower 


11.4 


11.8 


10.8 


11 


10.9 


9.9 


12.3 


12 


77.7 


78.4 


76.8 


Diplomas/degrees earned 0 (%) 


High school diploma 


74.8 


75.1 


74.5 


GED certificate 


22.8 


22.2 


23.5 


Occupational/technical certificate 


15.5 


17.0 


13.4 * 


Associate's degree or higher 


0.7 


0.8 


0.6 


None of the above 


1.5 


1.6 


1.4 


Date of high school graduation/GED certificate receipt (%) 


During the past year 


5.2 


5.3 


5.0 


Between 1 and 5 years ago 


19.5 


19.2 


19.9 


Between 5 and 10 years ago 


28.3 


28.3 


28.2 


More than 1 0 years ago 


47.1 


47.2 


46.9 


Main reason for enrolling in college 0 (%) 


To complete a certificate program 


6.6 


6.2 


7.2 


To obtain an associate's degree 


67.4 


68.3 


66.2 


To transfer to a 4-year college/university 


21.4 


21.6 


21.1 


To obtain/update job skills 


4.9 


4.2 


6.0 


Other 


2.1 


2.1 


2.2 


First person in family to attend college (%) 


30.7 


31.3 


29.7 


Highest degree/diploma earned by father 1 (%) 


Not a high school graduate 


23.4 


24.7 


21.4 


High school diploma or GED certificate 


46.4 


46.1 


46.9 


Some college or associate's degree 


20.2 


18.1 


23.1 ** 


Bachelor's degree or higher 


10.1 


11.1 


8.7 


Highest degree/diploma earned by mother 0 (%) 


Not a high school graduate 


17.5 


17.5 


17.5 


High school diploma or GED certificate 


42.3 


43.8 


40.0 


Some college or associate's degree 


30.5 


30.1 


31.0 


Bachelor's degree or higher 


9.7 


8.5 


11.5 * 


Language other than English spoken 


regularly in home (%) 


4.1 


3.7 


4.6 



Sample Size 1,329 782 547 

(continued) 



42 







Appendix Table A.l (continued) 

SOURCE: MDRC calculations using Baseline Information Form (BIF) data. 



NOTES: To analyze whether baseline characteristics jointly predicted research group status, a likelihood 
ratio test was performed. This yielded a p-value of 0.83. Convention suggests that this probability of 
differences occurring by chance is large enough that these differences can be ignored in the analyses. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels 
are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by campus. 

Characteristics shown in italic type are calculated for a proportion of the full sample and indicate 
nonexperimental data. Significance tests are not calculated for nonexperimental data. 

Missing values are not included in individual variable distributions. Any characteristic with more than 
5 percent of the sample missing a response is noted in the footnotes. 

Distributions may not add to 100 percent because of rounding. 

““Respondents who said they are Hispanic and chose a race are included only in the Hispanic category. 
Respondents who said they are not Hispanic and chose more than one race are considered multiracial. 
These respondents, combined with those who said they are American Indian/Alaskan Native or another 
race/ethnicity, are included under “Other.” 

b Benefits include unemployment/dislocated worker benefits, Supplemental Security Income (SSI) or 
disability, cash assistance or welfare, food stamps, and Section 8 or public housing. 

““Distributions may not add to 100 percent because categories are not mutually exclusive. 

d Excludes 22 percent of the sample who either did not know or declined to answer this question. 

““Excludes 9 percent of the sample who either did not know or decliend to answer this question. 
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About MDRC 



MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 
to learning what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness of so- 
cial and education policies and programs. 

Founded in 1974 and located in New York City and Oakland, California, MDRC is best known 
for mounting rigorous, large-scale, real-world tests of new and existing policies and programs. 
Its projects are a mix of demonstrations (field tests of promising new program approaches) and 
evaluations of ongoing government and community initiatives. MDRC’s staff bring an unusual 
combination of research and organizational experience to their work, providing expertise on the 
latest in qualitative and quantitative methods and on program design, development, implementa- 
tion, and management. MDRC seeks to leam not just whether a program is effective but also 
how and why the program’s effects occur. In addition, it tries to place each project’s findings in 
the broader context of related research — in order to build knowledge about what works across 
the social and education policy fields. MDRC’s findings, lessons, and best practices are proac- 
tively shared with a broad audience in the policy and practitioner community as well as with the 
general public and the media. 

Over the years, MDRC has brought its unique approach to an ever-growing range of policy 
areas and target populations. Once known primarily for evaluations of state welfare-to-work 
programs, today MDRC is also studying public school refomis, employment programs for ex- 
offenders and people with disabilities, and programs to help low-income students succeed in 
college. MDRC’s projects are organized into five areas: 

• Promoting Family Well-Being and Children’s Development 

• Improving Public Education 

• Raising Academic Achievement and Persistence in College 

• Supporting Low-Wage Workers and Communities 

• Overcoming Barriers to Employment 



Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local govern- 
ments, public school systems, community organizations, and numerous private philanthropies. 




